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1.  Introduction 


This  study  seeks  to  investigate  the  effects  of  level  of  information  (LOI)  on  human 
interaction  with  a  route-planning  agent  in  the  context  of  dismounted  infantry 
navigation.  Previous  research  has  shown  that  to  facilitate  effective  human-agent 
teaming  and  fluid  mixed-initiative  decision  making,  the  agent’s  user  interface  must 
facilitate  optimal  transparency  by  conveying  the  rationale  behind  its 
recommendations  but  must  do  so  without  burdening  the  human  with  an 
overwhelming  amount  of  data  (Lee  and  See  2004;  Fallon  et  al.  2010;  Lyons  2013). 

Lessons  learned  from  a  US  Navy  intelligent  autonomy  program  indicated  that 
human  operators  sometimes  questioned  the  accuracy  and  effectiveness  of  the  output 
produced  by  intelligent  systems,  such  as  those  generating  automated  plans,  due  to 
the  operators’  difficulties  in  understanding  the  rationale  behind  the  output 
(Linegang  et  al.  2006).  Lee  and  See  (2004)  recommended  that  when  feasible,  in 
order  for  the  operator  to  develop  appropriate  trust  and  reliance,  the  automated 
systems  must  convey  their  capabilities  and  limitations  to  the  operator.  Lee  (2012) 
proposed  that  to  increase  automation  transparency  to  the  operator,  system  designers 
should  make  the  system’s  3Ps  (purpose,  process,  and  performance),  and  the  history 
of  the  3Ps,  visible  to  the  operator.  However,  the  presentation  should  be  in  a 
simplified  form  (e.g.,  integrated  graphical  displays)  so  the  operator  is  not 
overwhelmed  by  the  amount  of  information  to  process  (Cook  and  Smallman  2008; 
Neyedli  et  al.  201 1).  In  service  to  this  goal,  proper  uses  of  information  visualization 
techniques  have  been  shown  to  help  operators  make  sense  of  information  and 
thereby  enhance  their  situation  awareness  of  the  mission/tasking  environments 
(Robertson  et  al.  2009). 

1.1  Cartography  and  Route  Planning 

Cartographic  researchers  are  currently  in  the  process  of  working  with  psychologists 
to  better  understand  how  humans  interpret  map  design.  Lorenz  et  al.  (2013) 
assessed  indoor  map  design  and  user  satisfaction.  They  explain  that  in  the  field  of 
cartography,  designers  try  to  provide  the  correct  context  to  allow  for  a  mental 
representation  and  linkage  between  the  user  and  the  map.  Providing  an  appropriate 
amount  of  visual  information  to  create  context  can  be  accomplished  by  adding 
visual  elements  to  increase  transparency.  Results  from  the  Lorenz  et  al.  (2013) 
study  show  that  the  helpfulness  of  a  landmark’s  presence  on  a  map  depends  on  an 
association  between  the  symbol  and  its  meaning  for  the  user.  The  same  study  also 
investigated  the  concept  of  route  complexity  in  terms  of  quantity  of  landmarks; 
however,  more  landmarks  did  not  increase  performance.  This  result  led  the 
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researchers  to  conclude  that  symbols  and  landmarks  may  overload  the  map  display 
in  a  complex  design,  while  an  appropriate  balance  of  symbols  and  landmarks  could 
produce  positive  results. 

Westerbeek  and  Maes  (2013)  examined  the  effect  of  visual  clutter  of  information 
on  map  designs  of  New  York  City  and  how  it  affected  the  ability  to  use  landmarks 
to  navigate.  They  found  that  during  interaction  with  cluttered  maps,  participants 
would  more  often  miss  essential  information,  instead  using  noncrucial  information 
in  their  way  finding.  Ware  (2012)  describes  this  extra  information  as  a  visual 
distractor.  Visual  distractors  interfere  with  the  brain’s  preattentive  processing, 
thereby  increasing  the  time  to  find  information  (Treisman  1985).  One  of  the  key 
psychological  functions  of  a  graphic  is  to  draw  attention  to  the  important  elements 
in  a  design  (Clark  and  Lyons  2010).  By  drawing  attention  to  the  important  items  in 
a  display,  divided  attention  is  reduced.  When  exploring  navigation  of  large  sets  of 
data,  users’  comprehension  is  best  when  they  have  to  focus  on  a  small  area  (Herman 
et  al.  2000).  Mosier  et  al.  (2007),  citing  Wickens  and  Flach  (1988),  explain  how  an 
operator  will  focus  on  salient  information  under  time  pressure,  but  without  salient 
information,  the  operator  could  become  lost.  The  concept  of  determining  what  is 
important  and  what  is  not  is  also  consistent  with  how  Kilgore  and  Voshell  (2014) 
built  designs  with  explicit  visual  cues  (such  as  icon  size  and  opacity)  to  help 
operators  recognize  the  key  variables.  Without  essential  information  indicators, 
operators  have  difficulty  building  a  complete  understanding  of  the  navigation 
information.  Instead,  simple,  clear  designs  have  been  shown  to  support 
understanding  (Lorenz  et  al.  2013). 

1.2  Decision  Making  and  Route  Planning 

Recent  research  by  Van  Tilburg  and  Igou  (2014)  demonstrated,  through  a  series  of 
wayfinding  experiments,  a  construct  known  as  action  continuation.  Once 
individuals  have  made  a  particular  choice  down  a  route,  they  tend  to  continue  along 
similar  pathways  rather  than  selecting  a  different  route.  There  are  several  strategies 
people  use  when  selecting  routes:  the  longest  straight  path  (Bailenson  et  al.  2000); 
the  choice  with  the  least  deviation  to  their  original  goal  (Hochmair  and  Frank  2000); 
the  hill-climbing  strategy,  where  they  break  the  route  into  a  series  of  subgoals  and 
select  the  easiest  available  choice  (Robertson  2001,  as  cited  in  Van  Tilberg  and 
Igou  2014);  or  they  may  abstain  from  decision  making  altogether  when  the  cost  of 
developing  strategy  outweighs  the  benefits,  also  known  as  criterion  failure 
(MacGregor  et  al.  2001). 
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Gigerenzer  and  Gaissmaier  (2011)  make  the  case  that  in  the  absence  of  imposed 
rules  upon  which  to  base  their  decisions,  people  often  use  simple  heuristics.  These 
heuristics  can  reflect  competing  goals,  such  as  ignoring  a  piece  of  information  to 
save  a  particular  resource.  Therefore,  without  some  constraints  to  control  how  an 
individual  searches  or  prioritizes  information  in  a  display,  each  person  could  make 
different  decisions  on  what  heuristics  to  apply. 

Effective  interface  design  uses  the  display  to  help  the  user  understand  his/her 
choices  when  making  decisions,  usually  through  visual  cues  or  design  elements. 
These  visual  cues  combine  in  such  a  way  as  to  assist  the  user  in  a  process  called 
“sensemaking”.  Sensemaking  (Klein  et  al.  2006)  is  a  deliberative  process  decision 
makers  use  to  improve  their  understanding  in  uncertain  and  ambiguous  situations. 
When  users  engage  in  sensemaking,  they  create  a  mental  schema  (or  frame)  as  they 
begin  to  understand  events  and  the  connections  between  events,  and  their 
perspective  is  continuously  updated  by  their  experiences.  In  the  case  of  this  study, 
sensemaking  provides  a  system  to  help  operators  transition  from  having  no 
understanding  of  the  events  before  them  to  developing  relationships  between  the 
extant  data  to  provide  information  for  future  decisions.  Munya  and  Ntuen  (2007) 
explain  that  in  the  context  of  a  battlefield  environment,  sensemaking  provides  an 
opportunity  to  provide  linkages  (or  as  they  put  it,  “connecting  dots”)  to  information 
that  otherwise  might  be  separated  across  domains.  Sensemaking  also  has  an 
adaptive  quality  in  that  it  allows  the  user  to  learn  as  they  go,  potentially  dealing 
with  unexpected  events  that  occur  along  the  way.  The  mental  model  that  the 
sensemaking  helps  to  shape  is  also  dynamic  in  nature,  which  explains  the 
modification  of  understanding  during  the  route  navigation  task  (Munya  and  Ntuen 
2007).  For  example,  in  a  route-planning  task,  a  user  begins  to  build  an 
understanding  of  what  the  different  icons  and  status  changes  mean  as  they  interact 
with  the  user  interface.  As  they  continue  to  learn,  they  can  optimize  their  choices 
to  achieve  mission  objectives. 

1.3  User  Interfaces  and  Information  Visualization 

A  potential  way  to  present  information  to  users  without  increasing  their  workload 
is  through  information  visualization.  Information  visualization  takes  existing 
performance  data  and  translates  it  into  a  graphical  format  that  can  be  more  easily 
understood.  Parsons  et  al.  (1999)  examined  over  150  studies  on  warnings  and  risk 
perceptions,  and  their  analysis  provided  a  series  of  recommendations  for  effective 
design.  Although  not  all  of  their  recommendations  apply  to  the  paradigm  of  route 
planning,  the  following  are  relevant: 
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•  Border  usage:  Increasing  the  weight  of  a  border  around  an  object  can 
increase  the  urgency  of  a  piece  of  information. 

•  Brevity:  Keeping  all  information  as  simple  and  as  clear  as  possible. 

•  Consequence  information:  Clarifying  the  consequences  of  each  choice 
facilitates  choosing  among  options. 

•  Visual  design:  Presenting  information  in  terms  of  bulleted  lists,  easy  to 
understand  pictograms,  and  alternative  label  designs  can  influence  the 
effectiveness  of  information  (Parsons  et  al.  1999). 

Complex  environments  are  made  simpler  by  appropriately  matching  the 
information  presentation  with  the  type  of  information  processing  required  to 
successfully  complete  the  task  (Vessey  1991).  For  example,  when  comparing  total 
values  of  noncategorical  data  between  groups,  the  most  effective  graph  is  a 
clustered  bar  graph  (Helfman  and  Goldberg  2007).  However,  some  research  has 
indicated  that  users  prefer  their  information  to  be  presented  as  short  textual  displays 
and  graphical  hints  that  appear  when  needed  and  then  disappear  afterwards  (Jepson 
et  al.  2009).  This  study  investigates  which  of  these  methods  is  not  only  preferable 
but  also  more  effective  in  successful  completion  of  the  route-planning  task. 

1.4  Trust  and  User  Interfaces 

The  amount  a  user  trusts  an  interface  can  directly  affect  their  willingness  to  use  it, 
their  performance,  and  how  they  respond  to  unexpected  scenarios  (Lee  and  See 

2004) .  The  higher  the  level  of  autonomy  of  the  system,  the  more  important  the 
system’s  LOI  becomes  in  fostering  trust  in  the  human  operator  (Wang  et  al.  2009). 
Consequently,  the  presentation  of  information  in  an  unfamiliar  format  reduces 
automation  transparency  (Kim  and  Hinds  2006).  This  study  investigates  how  the 
appropriate  LOI  and  the  preferred  manner  in  which  the  information  is  displayed 
affects  performance  and  trust  in  the  route-planning  agent. 

There  are  2  major  types  of  trust:  dispositional  and  history-based  (Merritt  and  Ilgen 
2008).  Dispositional  trust  is  a  stable  construct  describing  someone’s  feelings  about 
something  before  any  actual  encounter,  in  this  instance,  how  one  feels  about 
working  with  a  remote  monitoring  and  communications  system.  Dispositional  trust 
is  generated  by  exposure  to  a  variety  of  sources,  primarily  social  influences  such  as 
media  and  literature,  and  can  vary  widely  among  individuals  (Hancock  et  al.  2011). 
History-based  trust  is  trust  developed  from  direct  interaction  with  a  system.  It  is 
composed  of  4  facets:  competence,  predictability,  reliability,  and  faith  (Master  et  al. 

2005) .  As  an  individual’s  experience  working  with  a  particular  system  grows,  they 
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calibrate  their  trust  in  the  system  to  an  appropriate  level  that  optimizes  performance 
(Fallon  et  al.  2010).  The  effect  of  dispositional  trust  in  automation  on  performance 
and  of  LOI  on  history-based  trust  will  be  explored  in  this  study. 

1.5  Current  Study 

In  this  study,  the  participant  supervised  2  dismounted  Soldier  teams  from  a  remote 
location.  The  participant  directed  one  Soldier  team  as  to  which  route  they  should 
take  as  they  made  their  way  from  checkpoint  to  checkpoint  through  an  urban 
environment.  At  each  of  these  checkpoints  (hereafter  referred  to  as  decision  points 
[DPs]),  the  simulation  paused,  and  the  participant  received  information  regarding 
the  resource  demands  of  several  route  choices.  The  LOI  was  varied  across 
participants.  After  reviewing  the  available  information,  the  participant  selected  the 
next  route  segment,  and  the  simulation  would  resume.  The  participants’  goal  was 
to  manage  the  available  resources  so  that  the  Soldier  team  arrived  at  its  final 
destination  with  at  least  the  specified  minimum  amount  of  each  resource. 

In  Experiment  1,  the  participant  managed  Soldier  Energy  Units  (SEUs),  with 
instructions  they  must  have  at  least  5  SEUs  remaining  when  they  reached  the  final 
destination.  In  Experiment  2,  a  robotic  asset  was  added  to  the  Soldier  team,  so  in 
addition  to  managing  SEUs,  participants  also  managed  Robot  Battery  and  Robot 
Fuel,  with  the  requirement  they  must  have  at  least  10  units  of  each  when  they 
reached  the  final  destination. 

Between  DPs,  the  participant  monitored  the  environment  surrounding  the  second 
Soldier  team  as  it  progressed  through  the  simulation,  identifying  threats  defined  as 
armed  civilians  by  clicking  on  them  with  the  mouse.  The  participants  received  no 
feedback  on  this  task,  and  performance  on  this  secondary  task  was  not  evaluated, 
as  it  was  intended  solely  to  keep  the  participant  occupied  between  DPs. 

1.5.1  Study  Objective 

This  study  manipulated  the  LOI  the  operator  received  to  base  decisions  in  a  route¬ 
planning  task.  Participants  were  assigned  1  of  3  LOIs:  Low  LOI  provided  the 
operator  with  a  map  showing  3  route  segments,  color-coded  to  indicate  difficulty 
(which  implied  resource  usage);  Medium  LOI  provided  the  same  map  as  in  Low, 
and  displayed  the  resource  usage  requirements  of  each  route  segment  both  in  a  text 
box  and  in  a  bar  graph  below  the  map;  High  LOI  included  all  the  information  in  the 
Medium  condition  but  also  supplied  another  bar  graph  showing  predicted  resource 
usage  information  for  the  entire  remaining  route  (mission  completion).  Therefore, 
the  principal  objectives  of  this  study  were  to  1)  determine  how  LOI  supports 
operator  performance  and  2)  determine  how  the  addition  of  a  robotic  asset  and 
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managing  the  additional  resources  associated  with  such  affects  operator 
performance.  In  addition,  which  visual  elements  the  operators  relied  upon  (or  found 
to  be  most  useful)  in  their  decision-making  strategy  were  discussed. 


1.5.2  Hypotheses 

Based  on  the  reviewed  literature,  the  following  hypotheses  were  developed: 

•  Objective  1:  How  does  LOI  support  operator  performance? 

o  HI:  As  LOI  increases,  operator  trust,  as  measured  by  posttest  trust, 
will  increase. 

o  H2:  As  LOI  increases,  time  spent  at  each  DP  will  increase. 

o  H3:  As  LOI  increases,  total  decision  time  (DT)  will  also  increase. 

o  H4:  Soldier  energy  usage  will  decrease  as  LOI  increases  due  to  the 
operator  choosing  more  optimal  routes  when  supplied  more 
information. 

•  Objective  2:  How  does  having  additional  information  from  robotic 
asset  change  operator  performance? 

o  H5:  Soldier  energy  usage  will  be  greater  in  Experiment  2  than  in 
Experiment  1  due  to  the  higher  cognitive  load  associated  with  the 
additional  resource  management  duties. 

o  H6:  Robot  Battery  and  Robot  Fuel  usage  will  decrease  as  LOI 
increases  due  to  the  operator  choosing  more  optimal  routes  when 
supplied  with  more  information. 

o  H7:  Time  spent  at  each  DP  will  be  greater  in  Experiment  2  than  in 
Experiment  1. 

o  H8:  Total  DT  will  be  greater  in  Experiment  2  than  in  Experiment  1. 

•  Objective  3:  Across  the  2  experiments,  which  visual  elements  do 
operators  report  relying  upon  most  in  their  decision-making  strategy? 

o  H9:  As  LOI  increases,  operators  will  rely  upon  information  sources 
that  better  support  their  task  objectives  (i.e.,  text  box  and  bar 
graphs). 

o  H10:  As  LOI  increases,  operators  that  rely  upon  appropriate 
information  sources  (i.e.,  text  box  and  bar  graphs)  will  use  less 
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resources  than  those  that  rely  upon  unsuitable  information  sources 
(i.e.,  colored  map  routes  and  resources  gauges). 

2.  Method 


2.1  Participants 

A  total  of  120  participants  were  included  for  analysis  in  both  experiments.  Each 
experiment  included  60  participants  (20  per  group,  3  groups).  Experiment  1 
included  23  female,  34  male,  and  3  no  gender  reported  participants  (Minage  =18, 
Maxage  =  30,  Mage  =  21.20,  and  SDage  =  2.80),  while  Experiment  2  included  35 
female,  24  male,  1  no  gender  reported  participants  (Minage  =18,  Maxage  =  39,  Mage 
=  21 .58,  and  SDage  =  3.70).  In  Experiment  1,  one  participant  had  to  be  removed  due 
to  incomplete  data,  and  another  was  added  to  equal  out  the  conditions.  Participants 
were  recruited  through  the  University  of  Central  Florida’s  Institute  for  Simulation 
and  Training’s  Sona  System,  a  research  participant  management  system. 
Participants  received  payment  ($15/h)  as  compensation. 

2.2  Apparatus 


2.2.1  Hardware 

The  simulation  was  presented  on  a  Dell  personal  computer  with  a  22-inch  monitor, 
standard  mouse,  and  keyboard.  The  system  was  an  Intel  i7  3820,  with  a 
GIGABYTE  motherboard,  NVidia  GeForce  graphics  card  with  2-GB  random- 
access  memory  (RAM),  running  Windows  7.  The  system  had  16  GB  of  RAM. 

2.2.2  Simulator 

The  simulator  for  this  study  was  a  modified  version  of  the  Mixed  Initiative 
Experimental  Testbed  (Barber  et  al.  2008).  This  distributed  simulation  environment 
was  developed  for  investigating  how  automation  affects  human  operator 
performance.  The  study  interface  was  developed  using  the  General  Middle  Eastern 
environment,  and  the  target  detection  task  was  modelled  after  a  similar  task  used  in 
the  RoboLeader  studies  (Chen  et  al.  2011;  Chen  and  Barnes  2012;  Wright  et  al. 
2013).  The  user  interface  (Fig.  1)  displayed  a  virtual  environment,  time,  resource 
usage  gauges,  and  information  regarding  route  resource  requirements  (shown  only 
when  the  simulation  is  paused  at  DPs). 
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Fig.  1  Experiment  1  interface,  displaying  typical  information  presented  at  a  DP  for  the 
high  LOI  condition 

2.3  Surveys  and  Tests 

Participants  completed  a  demographics  questionnaire  at  the  beginning  of  the 
training  session  (Appendix  A).  Information  on  participants’  age,  gender, 
educational  level,  computer  familiarity  and  usage,  and  gaming  experience  was 
collected.  An  Ishihara  Color  Vision  Test  (using  9  test  plates)  was  administered  via 
PowerPoint  presentation.  All  participants  had  normal  color  vision. 

A  modified  version  of  the  “Trust  Between  People  and  Automation”  survey  (Jian  et 
al.  2000)  was  used  to  assess  participants’  dispositional  trust  in  automation 
(Appendix  B).  Operationalized  as  “Pretest  Trust”  for  analysis,  this  measure  was 
used  to  compare  individual  differences  in  performance  based  on  preexisting 
attitudes  regarding  automation.  Participants  were  categorized  based  on  a  quartile 
split  of  all  scores.  Experiment  1  revealed  the  following:  ( Min  =  2.55,  Q2  =  4.75, 
Q3  =  5.27,  Max  =  7.00,  M  =  5.25,  and  Mdn  =  5.27);  LowT  ( N  =  15,  score  less  than 
4.75),  MedT  (N  =  13,  score  greater  than  4.75  but  less  than  5.27),  or  HighT  (N  =  32, 
score  greater  than  5.27).  Experiment  2  revealed  the  following:  ( Min  =  2.82, 
Q2  =  4.55,  Q3  =  5.36,  Max  =  6.73,  M  =  5.18,  and  Mdn  =  5.36);  LowT  (N=  16, 
score  less  than  4.55),  MedT  (N  =  13,  score  greater  than  4.55  but  less  than  5.36),  or 
HighT  (N  =  31,  score  greater  than  6.73). 

Participants’  perceived  trust  in  the  route-planning  agent  was  evaluated  using  the 
Usability  and  Trust  Survey  from  Chen  and  Barnes  (2012)  (Appendix  C). 
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Operationalized  as  “Posttest  Trust,”  scores  from  the  Usability  and  Trust  Survey 
were  used  to  evaluate  how  differing  levels  and  presentations  of  information  affected 
their  perception  of  usability  and  trust  in  the  route -planning  agent.  Participants  were 
categorized  based  on  a  quartile  split  of  all  scores.  Experiment  1  showed  the 
following:  (Min  =  2.42,  Q2  =  5.00,  Q3  =  5.63,  Max  =  7.00,  M  =  5.61,  and 
Mdn  =  5.63);  LowT  (N  =  11,  score  less  than  5.00),  MedT  (N  =  19,  score  greater 
than  5.00  but  less  than  5.63),  or  HighT  (N  =  30,  score  greater  than  5.63).  Experiment 
2  revealed  the  following:  (Min  =  3.58,  Q2  =  4.29,  Q3  =  5.46,  Max  =  6.83,  M  =  5.28, 
and  Mdn  =  5.46);  LowT  (N  =  15,  score  less  than  4.29),  MedT  (N  =  15,  score  greater 
than  4.29  but  less  than  5.46),  or  HighT  (N  =  30,  score  greater  than  5.46). 

2.4  Measures  and  Variables 

2.4.1  Independent  Variables 

The  3  LOIs  for  Experiment  1  were  the  following: 

•  Low  LOI:  The  routes  were  color-coded  red/yellow/green  to  denote  relative 
energy  and/or  time  usage  requirements  required  to  traverse  that  section  of 
the  route.  Red  indicates  high  energy,  long  time,  or  both  high  energy  and 
long  time;  yellow  indicates  medium  energy,  medium  time,  or  both  medium 
energy  and  medium  time;  and  green  indicates  low  energy,  short  time,  or 
both  low  energy  and  short  time. 

•  Medium  LOI:  In  addition  to  the  LOI  of  the  Low  condition,  text  boxes 
containing  the  specific  energy  expenditure  and  time  to  complete  for  a 
segment  were  supplied.  A  bar  graph  was  also  presented  below  the  map  to 
facilitate  comparing  resource  requirements  across  route  selections. 

•  High  LOI:  In  addition  to  the  LOI  of  the  Medium  condition,  a  bar  graph 
showing  “predictive”  information  as  to  how  specific  route  choices  impact 
overall  resource  usage  for  mission  completion  was  supplied  (see  Lig.  1  for 
High  LOI  display). 

The  3  LOIs  for  Experiment  2  were  the  same  as  in  Experiment  1;  however,  resource 
information  for  the  robotic  asset  (Robot  Battery  and  Robot  Luel)  was  added  to  all 
charts  and  graphs  (see  Lig.  2  for  Experiment  2,  High  LOI  display). 
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Fig.  2  Experiment  2  interface,  displaying  typical  information  presented  at  a  DP  for  the 
high  LOI  condition.  ASM,  the  robotic  asset,  stands  for  autonomous  squad  member 

2.4.2  Dependent  Variables 

The  following  measures  were  collected  in  Experiments  1  and  2: 

•  Posttest  Trust:  evaluated  using  the  Usability  and  Trust  Surveys. 

•  Time  spent  at  DPs:  The  amount  of  time  (as  measured  by  the  system)  the 
participant  took  to  make  route  choice  decisions.  Both  the  time  spent  at  each 
DP  and  total  time  at  DPs  were  collected. 

•  Total  SEUs  used:  The  amount  of  energy  the  Soldier  used  to  move  along  the 
route. 

For  Experiment  2,  the  following  measures  were  added  to  accommodate  for  the 
addition  of  the  robotic  asset: 

•  Total  Robot  Battery  used:  This  was  measured  as  the  amount  of  Robot 
Battery  the  robot  used  as  it  moved  along  the  route. 

•  Total  Robot  Fuel  used:  This  was  measured  as  the  amount  of  Robot  Fuel  the 
robot  used  as  it  moved  along  the  route. 
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2.5  Procedure 


Experiment  1 

Upon  arrival,  the  participants  were  instructed  on  the  purpose  of  the  study  and  signed 
the  informed  consent  form.  Participants  then  completed  the  demographics 
questionnaire,  the  “Trust  Between  People  and  Automation”  survey,  and  a  brief 
Ishihara  Color  Vision  Test. 

The  participants  were  then  randomly  assigned  to  1  of  the  3  experimental  groups 
and  trained  for  their  experimental  task.  The  training  was  self-paced  and  delivered 
via  PowerPoint  slides.  The  slides  explained  the  experimental  scenario,  the 
information  they  would  receive,  their  goals  for  successful  mission  completion,  and 
steps  for  completing  various  tasks.  After  completing  the  training  slides,  participants 
underwent  a  training  scenario  that  mimicked  the  activities  that  would  occur  during 
the  experiment.  The  training  session  lasted  approximately  15  min,  and  all 
participants  demonstrated  adequate  mastery  of  the  system  before  proceeding  with 
the  study. 

The  experimental  session  lasted  about  30  min.  During  the  scenario,  participants 
guided  a  dismounted  Soldier  through  an  urban  environment  through  10  DPs.  The 
objective  was  to  arrive  at  the  final  destination  with  at  least  5  SEUs  remaining  in  as 
little  time  as  possible.  At  each  DP,  the  simulation  paused  and  the  participant 
received  information  regarding  the  resource  demands  of  3  different  routes  that 
could  be  used  to  proceed  to  the  next  DP.  Low  LOI  condition  participants  viewed  a 
map  showing  the  3  routes,  which  were  color-coded  to  indicate  difficulty  or  resource 
requirements.  Medium  LOI  participants  viewed  the  color-coded  map  but  also  had 
a  text  box  listing  the  energy  and  time  requirements  for  each  route  and  the  same 
information  displayed  on  a  bar  graph  below  the  map.  High  LOI  participants  had  all 
the  information  as  the  Medium  condition  but  also  had  an  additional  bar  graph 
depicting  projected  resource  requirements  if  they  continued  to  select  routes  that 
used  up  the  same  amounts  of  time  and  energy  as  the  ones  presently  displayed  on 
the  map.  Upon  completion  of  the  simulation,  participants  completed  the  Usability 
and  Trust  Survey.  During  debriefing,  participants  were  asked  about  which  sources 
of  information  they  used  in  the  route-planning  task,  then  the  experimenter  answered 
any  questions  they  had,  and  they  were  dismissed. 

Experiment  2 

Experiment  2  followed  the  same  procedure  as  Experiment  1,  except  a  robotic  entity 
(i.e.,  autonomous  squad  member  [ASM])  was  added  to  the  dismounted  team,  and 
the  resource  requirements  for  the  robot  entity  (Robot  Battery  and  Robot  Luel)  were 
displayed  in  addition  to  the  SEUs  and  Time  in  the  text  box  and  bar  graphs. 
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Therefore,  in  making  the  route- selection  decision  at  each  DP,  the  participants  had 
to  consider  additional  information  related  to  the  robot. 


3.  Results 


Between-subjects  analysis  of  variance  were  used  to  evaluate  the  effect  of  LOI  on 
the  dependent  variables,  a  =  .05.  Linear  regression  was  used  to  evaluate  predictive 
relationships  between  variables.  The  Chi-Square  Test  for  Independence  was  used 
to  evaluate  the  relationship  between  pretest  trust,  posttest  trust,  and  LOI,  a  =  .05. 
See  Table  1  for  means  of  performance  measures  across  LOI  for  both  experiments. 

Table  1  Means  and  SD  for  performance  measures  across  LOP 


Experiment 

Measure 

Low  LOI 

Med  LOI 

High  LOI 

1 

Total  decision  time  (s) 

85.26 

(32.38) 

103.28 

(41.49) 

103.28 

(32.59) 

SEU  used 

77.18 

(5.22) 

79.69 

(5.19) 

77.37 

(5.59) 

Total  decision  time  (s) 

91.59 

(50.94) 

158.36 

(71.77) 

183.39 

(73.90) 

9 

Soldier  energy  used 

81.93 

(6.25) 

82.43 

(4.68) 

82.52 

(3.81) 

z. 

Robot  Battery  used 

86.32 

(4.48) 

84.21 

(4.07) 

84.71 

(2.95) 

Robot  Fuel  used 

88.73 

(5.11) 

85.49 

(3.80) 

87.64 

(3.49) 

“Values  shown  are  means.  SD  in  parentheses. 


3.1  Experiment  1 

Study  Objective  1:  How  does  LOI  support  operator  performance? 

HI:  As  LOI  increases,  operator  trust  as  measured  by  posttest  trust,  will  increase. 

Pretest  trust  was  evaluated  as  a  potential  covariate  for  posttest  trust  and  a  predictor 
of  task  performance.  Pretest  trust  was  a  significant  predictor  of  posttest  trust, 
X2  (4,  60)  =  1 1.41,  j d  =  0.022,  and  Cramer’s  V  =  0.308,  indicating  there  was  no  effect 
of  experiment  on  posttest  trust.  Pretest  trust  was  not  a  significant  predictor  of  total 
DT,  F( 2,  57)  =  0.52,  and p  =  0.600  (Fig.  3). 
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Total  Decision  Time  by  Pre-Test  Trust  Group 
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Fig.  3  Total  decision  time  by  pretest  trust  group  membership.  Bars  denote  standard 
error  (SE). 

DT  at  DP  was  evaluated  to  see  if  it  predicted  posttest  trust  group.  Linear  regression 
indicated  that  DT  at  DP  was  not  a  significant  predictor  of  posttest  trust  for  the  Low 
or  the  Medium  LOI  conditions;  however,  it  was  a  significant  predictor  of  posttest 
trust  for  the  High  LOI  condition  (Table  2). 

Table  2  Regression  results  for  DT  at  DP  prediction  of  posttest  trust 


Condition 

R2 

Adj.  R2 

SEof 

the  estimate 

B 

SE  of 

B 

B 

t(18) 

P 

Low  LOI 

0.010 

-0.045 

1.138 

0.003 

0.008 

0.101 

0.432 

0.671 

Med  LOI 

0.036 

-0.017 

0.953 

-0.004 

0.005 

-0.191 

-0.824 

0.421 

High  LOI 

0.210 

0.166 

0.764 

-0.012 

0.005 

-0.458 

-2.187 

0.042 

Examining  the  High  LOI  condition  results,  it  appears  that  DT  at  DP  predicted  high 
postest  trust  scores  but  not  medium  or  low  posttest  trust  scores  (Fig.  4).  Examining 
the  regression  lines  for  all  trust  groups,  the  High  posttest  trust  group  had  R2  =  0.618, 
while  Medium  R2  =  0.044  and  Low  R2  =  0.035  (Fig.  4).  Overall,  participants  who 
scored  high  on  posttest  trust  spent  less  time  at  each  successive  DP  than  those  with 
medium  or  low  posttest  trust  scores. 
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High  LOI  Decision  Point  Times  by  Post-Test  Trust  Group 
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Fig.  4  Experiment  1  shows  regression  results  for  time  spent  at  DP  predicting  posttest  trust 
group  membership  for  the  high  LOI  condition  participants.  Symbols  denote  mean  DT  at  each 
DP  for  each  trust  group. 

H2:  As  LOI  increases,  time  spent  at  each  DP  will  increase. 

There  was  a  significant  interaction  between  LOI  and  DP  on  time  spent  at  the  DPs, 
Wilk’s  A  =  0.576,  F(18,  98)  =  1.73,  p  -  0.046,  and  np2  =  0.241  (Fig.  5).  Early  in  the 
simulation,  DPs  1-4,  participants  in  the  Medium  and  High  LOI  conditions  took 
longer  to  make  a  decision  than  those  in  the  Low  LOI  condition.  At  DP  5,  all 
participants  took  approximately  the  same  amount  of  time  to  reach  a  decision. 
Although  at  DPs  6  and  7  the  Low  LOI  participant’s  DTs  increased  slightly,  their 
average  time  for  DPs  6-10  continued  to  be  lower  than  their  Medium  and  High  LOI 
counterparts,  who  appeared  to  still  be  developing  a  strategy,  resulting  in  varied  DTs. 


Fig.  5  Experiment  1  showing  time  spent  at  each  DP,  sorted  by  LOI  condition 
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H3:  As  LOI  increases,  total  DT  will  also  increase. 


There  was  no  significant  difference  in  total  DT  between  the  3  LOI  conditions, 
F( 2,  57)  =  1.70,  p  =  0.193,  and  np 2  =  0.056  (Fig.  6).  Participants  in  the  Low  LOI 
condition  ( M  =  85.256)  spent  less  time  overall  making  decisions  than  either  the 
Medium  LOI  {M  =  103.276,  Cohen’s  d  =  0.484,  p  =  0.1 16)  or  High  LOI  conditions 
(M  =  103.282,  Cohen’s  d  =  0.555,  and  p  =  0.116),  but  not  significantly. 


Total  Decision  Time  by  LOI 
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Fig.  6  Experiment  1  with  total  DT  for  each  LOI  condition.  Bars  denote  SE. 


H4:  Soldier  energy  usage  will  decrease  as  LOI  increases,  due  to  the  operator 
choosing  more  optimal  routes  when  supplied  more  information. 


All  participants  completed  their  mission  with  at  least  5  SEUs  remaining. 
Participants  in  the  Medium  LOI  condition  used  more  SEUs  than  those  in  either  the 
Low  or  High  LOI  conditions,  though  not  significantly,  F( 2,  57)  =  1.36,  p  =  0.264, 
and  np2  =  0.046  (Fig.  7). 


Soldier  Energy  Used  by  LOI 
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Fig.  7  Experiment  1  showing  mean  total  SEUs  used  for  each  LOI  condition.  Bars  denote  SE. 


3.2  Experiment  2 


Study  Objective  1:  How  does  LOI  support  operator  performance? 

HI:  As  LOI  increases,  operator  trust,  as  measured  by  posttest  trust,  will  increase. 
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Pretest  trust  was  evaluated  as  a  potential  covariate  for  posttest  trust  and  a  predictor 
of  task  performance.  Pretest  trust  was  a  significant  predictor  of  posttest  trust, 
X2(4,  56)  =  14.20,  p  =  0.007,  Cramer’s  V  =  0.344,  indicating  there  was  no  effect  of 
experiment  on  posttest  trust.  Pretest  trust  was  not  a  significant  prediction  of  total 
DT,  F(2,  57)  =  1 .933,  and p  =  0. 154  (Fig.  8). 


Total  Decision  Time  by  Pre-Test  Trust  Group 
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Fig.  8  Experiment  2  shows  total  DT  by  pretest  trust  group  membership.  Bars  denote  SE. 

DT  was  evaluated  to  see  if  it  predicted  posttest  trust  group.  DT  per  DP  was  not  a 
significant  predictor  of  posttest  trust  for  any  LOI  condition  (Table  3).  There  is  no 
relationship  between  the  amount  of  time  spent  at  each  DP  and  the  posttest  trust 
scores. 

Table  3  Experiment  2:  regression  results  for  time  at  DP  prediction  of  posttest  trust 


Condition 

R2 

Adj.  R2 

SE  of  the 
estimate 

B 

SE  of  B 

P 

t(18) 

P 

Low  LOI 

0.077 

0.025 

0.892 

-0.005 

0.004 

-0.277 

-0.239 

0.238 

Med  LOI 

0.029 

-0.025 

1.054 

0.002 

0.003 

0.171 

0.737 

0.471 

High  LOI 

0.003 

-0.052 

1.085 

-0.001 

0.003 

-0.056 

-0.239 

0.814 

H2:  As  LOI  increases,  time  spent  at  each  DP  will  increase. 

There  was  not  a  significant  interaction  between  LOI  and  DP  on  time  spent  at  the 
DPs,  Wilks’  A=  0.616,  F(18,  98)  =  1.494,  p  =  0.108,  and  np2  =  0.215.  There  was  a 
significant  effect  of  LOI  on  the  time  spent  at  the  DPs,  Wilks’  A  =  0.450,  F( 9,  49) 
=  6.660,  p  <  0.001,  and  np2=  0.550  (Fig.  9).  Participants  in  the  Low  LOI  condition 
had  shorter  and  more  consistent  DTs  across  all  DPs  than  participants  in  either  the 
Medium  or  High  LOI  conditions.  There  was  no  significant  difference  in  average 
DTs  across  DPs  between  the  Medium  and  High  conditions.  Early  in  the  simulation, 
participants  in  the  High  LOI  condition  had  the  longest  time  at  DPs  1  and  2  and  then 
appeared  to  adopt  a  strategy  similar  to  those  in  the  Medium  LOI  condition  for  the 
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remainder  of  the  scenario.  From  DP  4  to  the  end,  Medium  and  High  LOI 
participants  continuously  reduced  their  time  spent  at  each  DP,  until  at  DP  10  their 
times  were  very  similar  to  those  in  the  Low  LOI  condition. 


Time  Spent  at  Decision  Point,  by  LOI 
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Fig.  9  Experiment  2  showing  time  spent  at  each  DP  for  each  LOI  condition 


H3:  As  LOI  increases,  total  DT  will  also  increase. 

There  was  a  significant  difference  in  total  DT  between  the  3  LOI  conditions, 
F( 2,  57)  =  10.228 ,p<  0.001,  and  np2  =  0.264  (Fig.  10).  Participants  in  the  Low  LOI 
condition  ( M  =  91.595)  spent  less  time  overall  making  decisions  than  either  the 
Medium  ( M  =  158.364,  Cohen’s  d  =  1.073,  and p  =  0.002)  or  High  ( M  =  183.385, 
Cohen’s  d  =  1.446,  and  p  <  0.001)  LOI  participants.  There  was  no  significant 
difference  in  total  DT  between  the  Medium  and  High  LOI  conditions  (Cohen’s 
d  =  0.343,  and p  =  0.238). 
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Fig.  10  Experiment  2  showing  the  total  DT  for  each  LOI  condition.  Bars  denote  SE. 
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H4:  SEU  usage  will  decrease  as  LOI  increases,  due  to  the  operator  choosing  more 
optimal  routes  when  supplied  more  information. 


All  participants  completed  their  mission  with  at  least  5  SEUs  remaining.  There  was 
no  significant  difference  in  total  SEU  used  between  conditions,  F( 2,  57)  =  0.080, 
p  =  0.924,  and  np2  =  0.003  (Fig.  11). 


Soldier  Energy  Used  by  LOI 
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Fig.  11  Experiment  2  showing  mean  total  SEU  used  for  each  LOI  condition.  Bars  denote 
SE. 


Study  Objective  2:  How  does  having  additional  information  from  robotic  asset 
change  operator  performance? 

H5:  SEU  usage  will  be  greater  in  Experiment  2  than  in  Experiment  1,  due  to  the 
higher  cognitive  load  associated  with  the  additional  resource  management  duties. 


There  is  a  significant  difference  in  total  SEUs  used  between  Experiment  1  and 
Experiment  2,  F(l,  118)  =  20.010,  p  <  0.001,  np2  =  0.145,  and  R 2  =  0.138  (Fig.  12). 
Overall,  total  SEU  use  was  greater  in  Experiment  2  than  in  Experiment  1 .  While 
the  actual  difference  between  the  2  experiments  is  not  great,  both  are  well  below 
the  95  SEU  maximum  allowed. 
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Fig.  12  Comparison  of  total  SEUs  used  between  Experiments  1  and  2.  Bars  denote  SE. 
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H6:  Robot  Battery  and  Robot  Fuel  usage  will  decrease  as  LOI  increases,  due  to  the 
operator  choosing  more  optimal  routes  when  supplied  with  more  information. 

There  was  no  significant  difference  in  total  Robot  Battery  used  between  conditions, 
F( 2,  57)  =  1.608,  p  =  0.209,  and  np2  =  0.053  (Fig.  13).  Robot  Battery  usage  was 
highest  in  the  Low  LOI  condition  ( M  =  86.315  and  SD  =  4.477)  than  in  either  the 
Medium  ( M  =  84.206,  SD  =  4.068,  p  =  0.091,  and  Cohen’s  d  =  0.493)  or  High 
(M=  84.710,  SD  =  2.949,  p  =  0.197,  and  Cohen’s  d  =  0.380)  LOI  conditions. 
However,  not  all  participants  completed  their  mission  with  at  least  10  Robot  Battery 
units  remaining.  Four  participants  in  the  Low  LOI  condition,  2  participants  in  the 
Medium  LOI  condition,  and  one  participant  in  the  High  LOI  condition  finished 
their  mission  with  less  than  10  Robot  Battery  units.  Removing  these  participants 
from  the  analysis  had  no  effect  on  the  outcome. 

Robot  Battery  Usage  by  LOI 

88.0 


Low  LOI  Med  LOI  High  LOI 

■  Mean  86.315  84.206  84.710 


Fig.  13  Experiment  2  showing  mean  Robot  Battery  usage  for  each  LOI  condition.  Bars 
denote  SE. 

There  was  a  marginally  significant  difference  in  total  Robot  Fuel  used  between 
conditions,  F( 2,  57)  =  3.090,  p  =  0.053,  and  np 2  =  0.098  (Fig.  14).  Robot  Fuel  usage 
was  higher  in  the  Low  LOI  condition  ( M  =  88.725  and  SD  =  5.114)  than  in  either 
the  Medium  (M  =  85.488,  SD  =  3.800,  p  =  0.018,  and  Cohen’s  d  =  0.719)  or  High 
(M  =  87.645,  SD  =  3.488,  p  =  0.419,  and  Cohen’s  d  =  0.247)  LOI  conditions.  Not 
all  participants  completed  their  mission  with  at  least  10  Robot  Fuel  units  remaining. 
Eight  participants  in  the  Low  LOI  condition,  3  participants  in  the  Medium  LOI 
condition,  and  2  participants  in  the  High  LOI  condition  finished  their  mission  with 
less  than  10  Robot  Fuel  units.  Removing  these  participants  from  the  analysis  had 
no  effect  on  the  outcome. 
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Robot  Fuel  Usage  by  LOI 
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Fig.  14  Experiment  2  showing  mean  Robot  Fuel  usage  for  each  LOI  condition.  Bars 
denote  SE. 


In  Experiment  2,  mission  LOI  condition  and  successful  mission  completion 
correlated  significantly,  y}{2)  =  10.050,  p  =  0.002,  and  Cramer's  V  =  0.409.  More 
participants  failed  to  complete  their  mission  with  the  required  minimum  resources  in 
the  Low  LOI  condition  than  in  either  the  Medium  or  High  LOI  conditions  (Lig.  15). 


Mission  Success  by  LOI 


Fig.  15  Experiment  2  showing  participant  success/fail  by  LOI  condition 

H7:  Time  spent  at  each  DP  will  be  greater  in  Experiment  2  than  in  Experiment  1. 

There  is  a  significant  difference  in  DT  at  each  DP  between  Experiment  1  and 
Experiment  2  (Table  4)  except  for  the  final  DP,  DP  10.  Time  to  make  a  decision 
was  significantly  longer  in  Experiment  2  than  in  Experiment  1.  This  difference  was 
greater  in  the  beginning  of  the  experiment,  with  the  difference  becoming  smaller  at 
later  DPs,  eventually  having  no  significant  difference  at  DP  10  (Fig.  16). 
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Table  4  Comparison  at  each  DP  between  Experiments  1  and  2 


DP 

F 

Sig. 

tip* 2 

Adj  R2 

Cohen’s  d 

DPI 

7.441 

0.007 

0.059 

0.051 

0.498 

DP2 

13.444 

0.000 

0.102 

0.095 

0.669 

DP3 

14.682 

0.000 

0.111 

0.103 

0.700 

DP4 

16.640 

0.000 

0.124 

0.116 

0.745 

DP5 

18.957 

0.000 

0.138 

0.131 

0.795 

DP6 

6.597 

0.011 

0.053 

0.045 

0.490 

DP7 

9.450 

0.003 

0.074 

0.066 

0.561 

DP8 

10.135 

0.002 

0.079 

0.071 

0.581 

DP9 

4.591 

0.034 

0.037 

0.029 

0.391 

DP10 

0.054 

0.817 

0.000 

-0.008 

0.042 
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19.44 

13.41 

18.41 

16.58 

12.90 

13.14 

13.01 

11.84 

9.40 

Fig.  16  Comparison  of  time  spent  at  each  DP  between  Experiments  1  and  2.  Bars  denote  SE. 


H8:  Total  DT  will  be  greater  in  Experiment  2  than  in  Experiment  1. 

There  is  a  significant  difference  in  total  DT  between  Experiment  1  and  Experiment 

2,  F(l,  118)  =  18.841,  p  <  0.001,  np2  =  0.138,  and  R2  =  0.130  (Fig.  17).  Overall, 
time  to  make  a  decision  was  significantly  longer  in  Experiment  2  than  in 
Experiment  1. 
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Fig.  17  Comparison  of  total  DT  between  Experiments  1  and  2.  Bars  denote  SE. 


There  is  a  significant  difference  in  DT  per  LOI  condition  between  Experiment  1 
and  Experiment  2  in  the  Medium,  F(l,  38)  =  8.831,  p  =  0.005,  and 
rip2  =  0.189,  and  High,  F(l,  38)  =  19.671,p  <  0.001,  and  np2  =  0.341,  LOI  conditions 
but  not  in  the  Low  LOI  condition,  F(l,  38)  =  0.221, p  =  0.641,  and  np2  =  0.006  (Fig. 
18).  Time  to  make  a  decision  increased  as  the  LOI  increased  more  in  Experiment  2 
than  in  Experiment  1. 
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Fig.  18  Comparison  of  total  DT  between  Experiments  1  and  2,  sorted  by  LOI  condition.  Bars 
denote  SE. 

Study  Objective  3:  Across  the  2  experiments,  which  visual  elements  do 
operators  report  relying  upon  most  in  their  decision-making  strategy? 

H9:  As  LOI  increases,  operators  will  rely  upon  information  sources  that  better 
support  their  task  objectives  (i.e.,  text  box  and  bar  graphs). 

This  question  is  aimed  at  understanding  the  information  attention  strategies  of  each 
of  the  operators  and  focuses  on  the  medium  and  high  levels  of  information  for  both 
Experiment  1  and  Experiment  2  (Fig.  19).  Examining  Experiment  1,  it  appeared 
that  most  operators  relied  on  the  resource  (bar)  graphs  (55%  in  medium  and  45% 
in  high),  instead  of  other  sources  of  information.  However,  as  the  LOI  increased, 
some  participants  shifted  back  to  the  map  route  color. 
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Fig.  19  Operator  preferred  information  sources  in  medium  and  high  LOI  conditions  for 
Experiments  1  and  2 

The  distribution  of  preferred  information  sources  in  Experiment  2  differed  from 
Experiment  1.  In  the  Medium  LOI  condition,  the  resource  graph  and  the  map 
information  boxes  were  preferred  and  utilized  equally,  while  the  map  route  colors 
and  resource  gauges  were  used  less  often.  In  the  high  condition,  the  distribution 
changed  once  again  with  the  increase  in  information;  reliance  on  the  resource 
graphs  dropped  while  resource  gauge  usage  increased. 

As  the  LOI  increases,  a  preferred  strategy  as  to  what  source  of  information  to  use 
for  task  completion  seems  to  disappear.  In  Experiment  1  Medium  condition,  the 
resource  graphs  were  the  preferred  information  source,  with  the  map  route  colors 
being  the  second  most  preferred.  However,  in  Experiment  1  High  condition,  fewer 
participants  chose  the  resource  graphs  as  their  preferred  source,  with  more  relying 
upon  the  map  route  colors  or  the  predictive  information.  Then  considering 
Experiment  2  Medium  condition  (which  is  arguably  the  next  highest  LOI  after 
Experiment  1  High  condition),  even  fewer  report  the  resource  graphs  or  the  map 
route  colors  as  the  preferred  information  source,  instead  relying  upon  the  map  info 
box  or  even  the  resource  gauges  to  base  their  decisions.  In  the  highest  LOI  condition 
(Experiment  2  High),  there  appear  to  be  4  equally  preferred  information  resources, 
indicating  that  there  is  a  limit  to  how  much  information  can  be  clearly  conveyed  in 
each  of  the  different  methods.  The  majority  of  participants  in  either  experiment  did 
not  consider  the  predicted  resource  usage  information  useful;  however,  this  could 
have  been  a  result  of  inadequate  training  as  to  how  to  utilize  that  particular  resource. 

H10:  As  LOI  increases,  operators  who  rely  upon  appropriate  information  sources 
(i.e.,  text  box  and  bar  graphs)  will  use  less  resources  than  those  who  rely  upon 
unsuitable  information  sources  (i.e.,  colored  map  routes  and  resources  gauges). 

In  Experiment  1  Medium  LOI,  there  was  a  significant  difference  in  SEU  usage  by 
information  source,  F(l,  18)  =  5.483,  p  =  0.031,  and  np2  =  0.233.  Operators  who 
used  the  text  box  and  resource  bar  graph  had  higher  SEU  usage  ( M  =  81.29  and 
SD  =5.02)  than  those  who  used  the  colored  routes  or  gauges  (M  =  75.95,  SD  =  3.59, 
and  Cohen’s  d  =  1.224).  In  the  High  LOI,  operators  who  used  the  text  box  and 
resource  bar  graph  had  lower  SEU  usage  (M  =  75.91  and  SD  =  4.71)  than  those  that 
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used  the  colored  routes  or  gauges  ( M  =  79.56,  SD  =  6.38,  and  Cohen’s  d  =  0.651); 
however,  this  did  not  reach  significance,  F(l,  18)  =  2.169,  p  =  0.158,  and 
np2  =  0.108. 

In  Experiment  2  there  was  no  significant  difference  in  SEU  usage  by  information 
source  in  either  the  Medium  LOI,  F(l,  18)  =  0.200,  p  =  0.660, and  np2  =  0.01 1,  or 
High  LOI,  F(\,  18)  =  0.011,  p  =  0.919,  and  np2  =  0.001,  conditions.  There  was  no 
significant  difference  in  Robot  Battery  usage  by  information  source  in  either  the 
Medium  LOI,  F(\,  18)  =  0.485,  p  =  0.495,  and  np2  =  0.026,  or  High  LOI, 
F(l,  18)  =  0.349,  p  =  0.562,  and  np2=  0.019,  conditions.  There  was  no  significant 
difference  in  Robot  Luel  usage  by  information  source  in  either  the  Medium  LOI,  F 
(1,  18)  =  0.019,  p  =  0.891,  lip2  =  0.001,  or  High  LOI,  F(l,  18)  =  0.026,  p  =  0.875, 
and  np2=  0.001,  conditions. 

One-third  of  the  participants  in  Experiment  2  failed  to  complete  their  mission 
successfully  (i.e.,  with  at  least  10  Robot  Battery  and  10  Robot  Luel  remaining). 
There  was  a  significant  difference  in  information  source  preference  between 
participants  who  successfully  completed  their  mission  and  those  who  did  not,  X 2 
(5,  N  =  60)  =  12.56,  p  =  0.028,  and  Cramer’s  V  =  0.458.  Often  participants  who 
failed  to  meet  mission  objectives  relied  upon  information  sources  that  were  not 
suitable  for  their  task  (Lig.  20).  The  addition  of  projected  information  appeared  to 
impact  their  performance  disproportionately,  as  evidenced  by  their  total  DT  in  the 
High  LOI  condition  (Lig.  21). 
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Fig.  20  Comparison  of  operator  preferred  information  source  between  participants  who 
failed  to  meet  mission  objectives  in  Experiment  2  and  those  who  were  successful 
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Fig.  21  Total  DT  for  Experiment  2,  sorted  by  success/fail  and  LOI  condition.  Participants 
who  failed  to  meet  mission  objectives  had  equal  or  shorter  decision  times  in  Low  and  Medium 
LOIs,  but  the  inclusion  of  the  predictive  information  in  high  LOI  resulted  in  total  DT  greater 
than  those  who  succeeded. 

4.  Discussion 


The  goals  of  this  study  were  to  determine  whether  and  how  differing  LOIs  of  a 
route-planning  agent  impacted  a  human  operator’s  decision  making  on  a  route- 
selection  task,  whether  the  addition  of  a  robotic  asset  reduced  performance,  and 
whether  LOI  affected  human  trust  of  the  agent.  In  all  experimental  conditions,  the 
task  was  to  make  a  tradeoff  decision  among  3  route  options,  each  with  unique 
constraints.  The  LOI  varied  across  experimental  conditions,  with  each  participant 
exposed  to  only  one  LOI. 

In  both  experiments,  participant’s  pretest  trust  scores  were  correlated  with  their 
posttest  trust  scores  but  were  not  predictive  of  DTs.  For  the  most  part,  DTs  did  not 
predict  posttest  trust  scores,  except  in  Experiment  1  High  LOI  where  DTs  at  each 
DP  were  predictive  of  high  posttest  trust  scores.  Participants  in  this  group  steadily 
decreased  the  time  spent  at  each  DP  (Fig.  4),  indicating  that  those  who  continuously 
learned  and  adapted  their  strategy  for  handling  the  large  amount  of  information 
ultimately  trusted  their  decisions  and  the  information  they  received  more  than  those 
who  did  not.  This  same  pattern  was  not  evident  in  Experiment  2  High  LOI  group, 
which  may  be  an  indication  that  there  can  be  so  much  information  that  the 
participant  cannot  make  sense  of  what  information  is  important  and  what  is  not 
(Westerbeek  and  Maes  2013). 

Participants  in  the  Low  LOI  condition  had  shorter  DTs  than  those  in  Medium  or 
High  LOI  conditions,  and  DTs  in  the  Low  LOI  were  similar  in  both  experiments. 
An  effect  of  LOI  on  DTs  becomes  apparent  in  the  Medium  and  High  LOI 
conditions,  particularly  for  DPs  1-5.  In  Experiment  1,  DTs  for  DPs  1-5  were 
similar  between  Medium  and  High  conditions;  however,  for  DPs  6-10,  DTs  varied 


Approved  for  public  release;  distribution  is  unlimited. 


25 


widely  between  the  2  conditions.  Participants  in  the  High  and  Medium  conditions 
seemed  to  find  a  point  where  they  felt  confident  in  their  ability  to  complete  their 
mission  successfully,  after  which  they  appeared  to  explore  alternate  strategies, 
resulting  in  varied  DTs  at  each  DP  for  the  remainder  of  the  experiment.  This  trend 
suggests  that  without  constraints  to  influence  their  decisions,  participants  will  vary 
their  strategy  based  on  other  means  besides  direct  experience  (Gigerenzer  and 
Gaissmaier  2011). 

Experiment  2  provided  these  additional  constraints  by  adding  a  robotic  asset. 
Participants  in  the  High  LOI  condition  started  with  the  longest  DTs  at  DPs  1  and  2, 
most  likely  due  to  the  additional  information  presented  in  the  predictive  resource 
usage  graphic.  They  seem  to  have  decided  to  disregard  this  additional  information 
as  part  of  their  strategy  development,  as  illustrated  by  their  DTs  being  similar  to 
those  in  the  Medium  condition  from  DP  3  onward.  After  DP  4,  DTs  for  Medium 
and  High  LOI  conditions  become  consistently  shorter,  evidence  of  strategy 
development  and  practice,  with  their  times  finally  approaching  DTs  similar  to  those 
of  the  Low  LOI  condition  in  DP  10.  This  trend  appears  to  be  an  indication  of 
sensemaking  in  progress;  operators  are  modifying  their  mental  model  of  the  agent 
as  they  gain  understanding  as  to  how  to  successfully  complete  their  mission  (Klein 
et  al.  2006).  Comparing  DTs  across  the  2  experiments,  participants  in  Experiment 
2  took  significantly  longer  than  those  in  Experiment  1,  in  both  terms  of  total  DT 
and  individual  DT,  and  this  difference  becomes  more  pronounced  as  the  LOI 
increases  (Lig.  16).  This  increased  time  supports  the  concept  that  without  clear 
indication  of  essential  information,  users  have  greater  difficulty  when  trying  to 
build  understanding  (Westerbeek  and  Maes  2013).  Lorenz  et  al.  (2013)  noted  that 
in  more  complex  displays,  screen  elements  (in  our  case  maps,  graphs,  and  gauges) 
might  overload  the  operator,  causing  an  inability  to  meet  appropriate  performance 
goals.  However,  the  data  indicate  that  most  participants  in  the  Medium  and  High 
LOI  conditions  had  better  resource  management  than  their  Low  LOI  counterparts, 
even  though  they  took  more  time  at  each  DP,  suggesting  that  taking  more  time  may 
not  necessarily  mean  an  “overload”  of  information. 

All  participants  in  both  experiments  successfully  met  their  performance  goal  of 
completing  the  task  with  at  least  5  SEUs  remaining;  however,  20  participants  in 
Experiment  2  failed  to  meet  the  robotic  asset  goals  of  completing  the  mission  with 
at  least  10  Robot  Battery  units  and  10  Robot  Luel  units  remaining.  The  majority  of 
participants  (N  =  12)  who  failed  to  meet  mission  objectives  were  in  the  Low  LOI 
condition,  indicating  that  there  is  a  minimum  amount  of  information,  or  a  preferred 
information  presentation,  required  to  successfully  balance  multiple  resources. 
Overall,  participants  in  the  Medium  and  High  LOI  conditions  managed  the  robotic 
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assets  more  effectively  than  those  in  the  Low  LOI  condition,  demonstrating  the 
importance  of  access  to  appropriate  information  for  the  task. 

Even  though  the  Soldier  movement  goal,  “complete  with  at  least  5  Soldier  Energy 
Units,”  was  more  difficult  to  achieve  than  the  robotic  asset  goals,  all  participants 
successfully  achieved  this  goal.  Even  though  no  specific  instructions  were  provided 
that  ranked  the  goals  in  order  of  importance,  all  participants  seemed  to  have 
independently  ranked  this  goal  as  more  important  to  achieve  than  the  others. 
Reviewing  the  average  SEU  used  in  both  Experiments  1  and  2,  regardless  of 
condition,  SEU  used  never  approach  the  maximum  95  allowed,  instead  averaging 
about  80  SEUs  across  both  experiments.  Access  to  higher  LOI  did  not  have  a 
significant  impact  on  SEU,  indicating  this  resource  usage  was  not  a  result  of  LOI 
or  information  presentation.  Prioritizing  this  goal,  even  when  evident  that  the  SEU 
goal  would  be  achieved  with  room  to  spare,  may  have  contributed  to  failure  on  the 
other  goals.  This  finding  could  have  important  ramifications  for  designing  effective 
human-agent  teams,  as  operators  favoring  human-based  goals  at  the  expense  of 
agent-based  goals,  even  when  there  is  no  danger  of  failure  on  the  human-based 
goals,  could  ultimately  lead  to  mission  failure. 

As  the  LOI  increased,  a  preferred  strategy  as  to  what  source  of  information  to  use 
for  task  completion  seemed  to  disappear.  In  Experiment  1  Medium  LOI  condition, 
participants  had  a  clear  preference  for  using  resource  graphs  as  a  primary  source  of 
information,  but  as  the  LOI  increased  to  High,  and  then  increased  again  in 
Experiment  2,  this  preferred  strategy  disappeared,  until  finally  in  Experiment  2 
High  LOI  we  see  no  clearly  preferred  resource  but  rather  4  equally  used  sources. 
Again,  without  clear  understanding  as  to  which  information  is  the  most  useful, 
participants  found  it  more  difficult  to  build  understanding  and  thus  one  clear 
strategy  never  emerged  (Westerbeek  and  Maes  2013).  This  finding  supports  the 
idea  that  it  is  important  to  keep  information  presentation  as  clear  and  simple  as 
possible  (Parsons  et  al.  1999). 

Although  this  research  had  findings  for  LOI  and  decision-making,  the  research  also 
had  some  limitations: 

•  Inclusion  of  workload  measurements  may  have  assisted  in  understanding 
when  or  if  the  user  was  struggling  with  too  much  information  to  consider 
while  making  decisions. 

•  Inclusion  of  individual  difference  factors  may  have  shed  light  on  differential 
effects  of  performance  and  information  visualization  methods  that  would 
help  in  the  design  of  more  universally  useful  interfaces. 
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5.  Conclusion 


In  the  current  study,  we  investigated  whether  differing  LOIs  in  a  route-planning 
agent  affected  an  operator’s  decision  making  on  the  route-selection  task,  the  impact 
of  adding  robotic  resources  to  the  team,  and  the  impact  of  trust  on  operator 
performance  through  2  experiments.  Overall,  increasing  LOI  increased  DT  for 
participants  and  decreased  the  likelihood  of  failure  to  complete  the  mission  within 
predefined  parameters;  however,  differential  effects  of  sensemaking,  DT,  and 
reported  trust  are  apparent.  Participants  in  Medium  and  High  LOIs  in  both 
experiments  continued  to  improve  DTs  throughout  the  scenario,  as  they  appeared 
to  revise  and  adapt  their  strategy,  improving  to  the  point  of  having  nearly  the  same 
DT  as  the  Low  LOI  participants  at  DP  10  in  Experiment  2.  This  learning  effect  was 
more  apparent  in  Experiment  2,  where  the  additional  resource  management 
requirements  appeared  to  have  imposed  constraints  that  helped  the  participants 
focus.  Participants  in  Experiment  1  High  LOI  who  demonstrated  continued  learning 
throughout  the  scenario  reported  higher  trust  in  the  system  than  those  who  adopted 
a  strategy  early  and  did  not  alter  it.  Thus,  DT  on  the  route-planning  task  depended 
not  only  on  the  amount  of  information  an  individual  had  to  consider,  but  how 
effective  they  were  at  learning  and  adapting  as  well  (Klein  et  al.  2006). 

All  participants  successfully  met  their  performance  goal  of  completing  their 
mission  with  at  least  5  SEUs  remaining.  However,  20  participants  in  Experiment  2 
failed  to  meet  the  robotic  asset  goals  of  completing  their  mission  with  at  least  10 
Robot  Battery  and  10  Robot  Fuel  remaining.  The  majority  of  participants  (N  =  12) 
who  failed  to  meet  mission  objectives  were  in  the  Low  LOI  condition,  which 
indicates  there  is  either  a  minimum  amount  of  information  or  an  optimal 
information  presentation  required  to  balance  multiple  resources  successfully. 
Overall,  participants  in  the  Medium  and  High  LOI  conditions  managed  the  robotic 
assets  more  effectively  than  those  in  the  Low  LOI  condition,  demonstrating  the 
importance  of  access  to  appropriate  information  for  the  task.  Even  though  no 
specific  instructions  were  provided  that  ranked  the  mission  objectives  in  order  of 
importance,  participants  seemed  to  have  independently  ranked  the  human-centric 
goal  of  completing  with  at  least  5  remaining  SEUs  as  more  important  to  achieve 
than  the  robot-centric  goal.  Prioritizing  the  human-centric  SEU  goal  over  the  robot¬ 
centric  goal,  even  when  it  is  evident  that  the  SEU  goal  would  be  achieved  with 
room  to  spare,  may  have  contributed  to  the  failure  on  the  robot-centric  goals.  This 
finding  could  have  important  ramifications  for  designing  effective  human-agent 
teams,  as  operators  favoring  human-centric  goals  at  the  expense  of  agent-centric 
goals,  even  when  there  is  no  danger  of  failure  on  the  human-centric  goals,  could 
ultimately  lead  to  mission  failure. 
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Examining  reported  information  resource  preferences  in  Experiment  1,  as 
information  increased,  strategies  changed  slightly  but  overall  remained  similar; 
however,  as  the  amount  of  information  continued  to  increase  in  Experiment  2, 
preferences  began  to  differ  as  each  participant  struggled  to  meet  their  objectives.  In 
Experiment  1  there  was  a  clear  preference  for  using  the  bar  graphs  to  make  route 
decisions  over  route  color-coding  and  text  information,  which  agrees  with  matching 
information  and  usage  to  appropriate  presentation  (Vessey  1991).  In  Experiment  1, 
when  the  LOI  increased  from  Medium  to  High,  some  participants  returned  to  the 
baseline  condition  (route  color-coding).  As  LOI  continued  to  increase  (Experiment 
2  Medium  LOI),  more  participants  abandoned  the  bar  graphs  for  other  information 
presentations,  and  in  the  highest  LOI  (Experiment  2  High),  there  is  no  longer  one 
preferred  information  source,  instead  splitting  into  4  (roughly)  equally  preferred 
sources.  Interestingly,  as  the  information  increased,  usage  of  graphs  was  reduced 
in  favor  of  simpler  displays,  such  as  gauges  and  text  boxes.  As  the  amount  of 
information  increases,  participants  struggled  to  make  sense  of  what  was  important, 
reverting  to  simpler  methods,  and  illustrating  the  importance  of  defining  essential 
information  in  complex  displays  (Westerbeek  and  Maes  2013). 

Future  research  could  focus  on  presentation  of  information  and  how  to  properly 
display  robotic  asset  constraints/resources  and  conveying  that  to  an  operator.  This 
line  of  research  would  require  a  clear  definition  of  essential  information,  with  the 
capability  to  adapt  given  changing  goals  and  priorities.  We  are  currently  in  the 
process  of  developing  multiple  studies  on  the  conveyance  of  information  in  human- 
agent  teams,  which  should  assist  in  further  defining  the  problem  space.  The  key 
contribution  of  this  work  demonstrates  that  there  is  a  limit  on  information 
processing  for  human-agent  teams  and  that  LOI  can  both  positively  and  negatively 
impact  completion  of  mission  goals. 
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Participant# _  Age _ Major _  Date _  Gender _ 

1.  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _  Other _ 

2.  When  did  you  use  computers  in  your  education?  (Circle  all  that  apply) 

Grade  School  Jr.  High  High  School  Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer?  ( Circle  all  that  apply) 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

How  often  do  you: 

Use  a  mouse?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  a  joystick?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Use  a  touch  screen?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  icon-based  programs/software? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  programs/software  with  pull-down  menus? 

Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 
Use  graphics/drawing  features  in  software  packages? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  E-mail?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Operate  a  radio  controlled  vehicle  (car,  boat,  or  plane)? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Play  computer/video  games? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

5.  Which  type(s)  of  computer/video  games  do  you  most  often  play  if  you  play  at  least  once  every  few  months? 

6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 

_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 

7.  Are  you  in  your  good/  comfortable  state  of  health  physically?  YES  NO 

If  NO,  please  briefly  explain: 


8.  How  many  hours  of  sleep  did  you  get  last  night? 


9.  Do  you  have  normal  color  vision? 

YES 

NO 

10.  Do  you  have  military  service? 

YES 

NO 

hours 


If  Yes,  how  long 
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Automation  Survey 


Automation  refers  to  a  system  that  reduces  the  need  for  human  work.  According  to 
Lee  and  See  (2004),  “Automation  is  technology  that  actively  selects  data, 
transforms  information,  makes  decisions,  or  controls  processes.”  Below  is  a 
statement  evaluating  your  feelings  about  automation.  Please  circle  the  number  that 
best  describes  your  feeling  or  impression. 

1  =  not  at  all;  7  =  extremely 

1.  Automation  is  deceptive. 

1  2  3  4  5  6  7 

2.  Automation  systems  behave  in  an  underhanded  manner. 

1  2  3  4  5  6  7 

3.  I  am  suspicious  of  the  intent,  action,  or  outputs  of  automation. 

1  2  3  4  5  6  7 

4.  I  am  wary  of  automation. 

1  2  3  4  5  6  7 

5.  The  actions  of  automated  systems  will  have  harmful  or  injurious 
outcomes. 

1  2  3  4  5  6  7 

6.  I  am  confident  in  automation. 

1  2  3  4  5  6  7 

7.  Automated  systems  provide  security. 

1  2  3  4  5  6  7 

8.  Automated  systems  have  integrity. 

1  2  3  4  5  6  7 

9.  Automated  systems  are  dependable. 

1  2  3  4  5  6  7 

9.  Automated  systems  are  reliable. 

1  2  3  4  5  6  7 

10.  I  can  trust  automated  systems. 

1  2  3  4  5  6  7 

The  Trust  Survey  is  based  on  the  questionnaire  of  Human-Computer  Trust  from  Jian  et  al.  (1998) 
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1.  The  Route  Planning  Agent  is  deceptive. 

Strongly  DISAGREE  — | — | — | — | — | — j  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

2.  The  Route  Planning  Agent  behaves  in  an  underhanded  manner. 

Strongly  DISAGREE  ~~| — |— -| - 1-—|  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


3.  I  am  suspicious  of  the  Route  Planning  Agent’s  intent,  action  or  outputs. 

Strongly  DISAGREE  — J . | — | — | — Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

4.  I  am  wary  of  the  Route  Planning  Agent. 

Strongly  DISAGREE  — | . | — | — | —  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


5. 


6. 


The  Route  Planning  Agent’s  actions  will  have  a  harmful  or  injurious 

Strongly  DISAGREE  — | — j — | — | — | — [  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

outcome. 

I  am  confident  in  the  Route  Planning  Agent. 

Strongly  DISAGREE  — — . | — | —  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


7.  The  Route  Planning  Agent  provides  security. 

Strongly  DISAGREE  — | . | — | — Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


8.  The  Route  Planning  Agent  has  integrity. 

Strongly  DISAGREE  — | — | — | — | — | — Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


9.  The  Route  Planning  Agent  is  dependable. 

Strongly  DISAGREE  — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


10.  The  Route  Planning  Agent  is  reliable. 

Strongly  DISAGREE  — | — | — | — | — | — Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

11. 1  can  trust  the  Route  Planning  Agent. 

Strongly  DISAGREE  — | — | — | — | — | — Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

12. 1  am  familiar  with  the  Route  Planning  Agent. 

Strongly  DISAGREE  — | — | — | — | — | — Strongly  AGREE  N/A 
1  2  3  4  5  6  7 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ASM 

autonomous  squad  member 

DP 

decision  point 

DT 

decision  time 

LOI 

level  of  information 

RAM 

random-access  memory 

SE 

standard  error 

SEU 

Soldier  Energy  Unit 
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